NVIDIA Details Process to Replicate MLPerf v5.0 Training Benchmarks for LLMs

NVIDIA has released a technical breakdown for reproducing training scores from the MLPerf v5.0 benchmarks, focusing on Llama 2 70B LoRA fine-tuning and Llama 3.1 405B pretraining. The company previously reported achieving up to 2.6x higher performance in these benchmarks, which evaluate machine learning model efficiency.

Hardware requirements are stringent: Llama 2 70B LoRA demands an Nvidia DGX B200 or GB200 NVL72 system, while Llama 3.1 405B requires at least four GB200 NVL72 systems with InfiniBand connectivity. Storage needs range from 300GB for LoRA fine-tuning to 2.5TB for full pretraining.

The recommended cluster setup leverages NVIDIA Base Command Manager with Slurm, Pyxis, and Enroot for environment management. Optimal performance requires RAID0-configured local storage and high-speed networking via NVLink and InfiniBand.